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1. INTRODUCTION 

Usability is one of the key success factors of a website that determines the ease or difficulty of users 
operating a website [1]-[3]. Besides the performance and reliability factors, usability is included as an 
important attribute of software quality [2]. According to International Organization for Standardization (ISO) 
9241-11, usability is defined as “the extent to which a system, product or service can be used by specified 
users to achieve specified goals with effectiveness, efficiency and satisfaction in a specified context of 
use” [4], [5]. Usability is related to user experience when interacting with the system including the aspect of 
learnability, understandability, and operability [6], [7]. The usability level of the software can be measured 
using an automated or manual approach using tools or evaluations from end users or experts. These 
approaches can identify usability issues in terms of efficiency, effectiveness, and user satisfaction. The 
choice of evaluation method greatly influences the accuracy of the results which will inform the quality of 
the measured software. Usability evaluation is not only needed as a reference for the quality of the website 
(the lower value will affect the efficiency of use and user satisfaction), but also contribute for decreasing the 
risk of the result [8]. 

The application’s interface is usually designed based on the perspective of the developer, so it is 
often not according to user needs [9]. Therefore, an evaluation of the user interface needs to be done to 
ensure that the interface is in accordance with user requirements. Based on the assessment mechanism, there 
are three categories of usability evaluation, namely: 1) User-based method, where the system user is involved 
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in the process of identifying usability issues. The purpose of this method is to measure the user performance 
or satisfaction when interacting with the system [10]; 2) Evaluator based method, where usability issues are 
identified by a group of evaluators. One is a heuristic evaluation that involves a group of experts to assess the 
user interface; and 3) Device-based method, where identification of problems related to the user interface is 
done using software [6]. The way this device works is to ensure that the system meets the specified usability 
standards. 

Heuristic evaluation is the most popular method used in software usability measurement because it 
is fast and inexpensive. The method proposed by Nielsen and Molich involves a group of evaluators 
(generally numbering 3 to 5 people) who understand some of the principles of usability (known as 
‘heuristics') or who are usability experts [11]. These ten principles are still relevant and are widely used in 
evaluating software interfaces. Heuristic evaluation gives effective results when applied to software 
development iteratively even though it does not involve many evaluators, it is able to detect problems related 
to usability at the software interface [11], [12]. Heuristic evaluation methods can be applied to web-based 
software [13], [14] and mobile [15] both applying qualitative and quantitative approaches. Website 
evaluation using heuristic evaluation has been carried out by Rangraz, et al. [16] where heuristic evaluation is 
applied to evaluate the national health information system with the number of evaluators as many as five 
experts who judge based on the ten principles of Nielsen's usability; Inal [14] used a heuristic method to 
inspect the national library website where the evaluators involved were 57 students then combined with the 
System Usability Scale (SUS) and Net Promoter Score (NPS) methods. The heuristic evaluation used is able 
to detect most of the crucial usability issues in the system. However, there are not many studies that discuss 
the application of heuristic evaluation on children's encyclopedia web. 

One of the heuristic evaluation methods that is comprehensive, empirical, and can be applied to 
websites with various types is the Sirius framework [17]. This framework covers ten aspects of assessment 
which includes general aspects, website identity and information, accomplishment of user interface, and 
search features. Furthermore, all aspects are then break down into 83 criteria [13], [17]. Compared with other 
quantitative heuristic evaluation methods [18]-[20], this framework can be used in evaluating several types of 
websites including educational websites, e-commerce, e-learning, institutions, government, and social media. 
In addition, the criteria are more comprehensive (representing all the principles of Nielsen's heuristic 
evaluation) and the results of the evaluation are quantitative values that state the level of website usability. 
Several studies have used this framework for evaluating usability including Aparna and Baseer [7] and 
Rodriguez et al. [21]. 

Anapedia is a web encyclopedia designed specifically for children [22]. The app was built because 
there are still very few Indonesian-language encyclopedias for children. Given the large number of school- 
age children in Indonesia, this website will be able to support the learning process of children. Anapedia Web 
has been tested using black-box testing to measure the performance aspects of the application. Based on the 
previous research, users still do not feel comfortable when adopting Anapedia because of the user interface. 
Even though, usability aspects have a strong connection with user readiness [23]. Therefore, in this study, we 
analyze the user interface of Anapedia from a perspective of an education website using the Sirius 
framework. The research questions in this study are: 1) What is the usability level of Anapedia's website 
according to Sirius framework evaluation; and 2) What are the usability problems of Anapedia. 


2. RESEARCH METHOD 

This section explains research methodology to answer research questions. The overall process is 
shown in Figure 1. The methodology is divided into three phases. First stage, a literature review process was 
carried out to obtain an overview of usability evaluation both manually and automatically in the usability 
evaluation of the children's encyclopedia website. Sirius has 10 assessment aspects (as shown in Table 1) and 
each aspect has sub-criteria so that the total criteria are 83 items [13]. Framework on Figure 2 is able to 
produce global usability values based on expert judgment while providing information about usability errors 
from websites based on their type. This information is useful for the website developer to improve the 
interface. 

The evaluators provide an assessment of all criteria covering 10 aspects as shown in Table 1. There 
are 2 rating scales, i.e: 1) using the numbers 0 to 10 where a value of 10 is given if the criterion is fully 
achieved and a value of 0 if it is not met or a value between 0 and 10 according to the level of achievement of 
the website; 2) using labelling as shown in Table 2, where the value of YES if a criterion is achieved by the 
web or not conform in the whole site (NWS) if the web does not meet the criteria or values other than both, 
namely not conform in the main links (NML), not sustained in the home page (NHP), and not conform in one 
or more subpages (NSP). Not applicable criteria (NA) if the evaluator decides that an assessment criteria 
cannot be applied to the web. Each criterion is given a weight (critical/major/moderate/minor) where the 
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weight of each criterion is determined by the type of website to be evaluated. Based on the value given by the 
evaluator, global usability can be calculated using (1), where PU: usability procentage (%), nec: number of 
criteria evaluated (maximum number is 83, but not all of them are used depending on the type of website), sv: 
Sirius value (value of evaluator), wc: weighting coefficient (weighting for criteria calculated using (2). For 
(2), rv is the weight value of each criterion. If the evaluator is more than one person, then the global usability 
value is obtained from the average calculation. 


PU= List necWCi*svi) y 100 (1) 


Li=1,neclwej*10) 
rvi 


we, = ——+— 
Yietnec(Tv;) 


(2) 


In the second stage, the usability evaluation process uses two approaches that begin with developing 
an evaluation scenario. For automatic assessment, it is done by entering the Anapedia web URL 
(http://anapedia.org) into the Google PageSpeed web [24]. The manual assessment begins by determining the 
evaluator who will provide an assessment based on the criteria in the Sirius framework. Based on research 
from [11], [13], the number of expert evaluators involved in heuristic assessment is as many as five people. 
However, the evaluation in this study will be carried out by seven experts as evaluators. Furthermore, the 
manual evaluation uses the following scenario [12]: 

— Review: Evaluators are given time to use the Anapedia website at http://anapedia.org/. For the search 
feature, the evaluator includes several keywords namely "photosynthesis”, "plants", "snake", "sun", and 
"rainbow" in the text field provided. 

— Evaluation: Evaluator observes search results. The evaluator evaluates the overall website interface based 
on 10 aspects in the Sirius framework. 

— Rate severity: Evaluators give ratings for each criterion using a rating scale of 0... 10 or NWS ... YES. 

The Anapedia web page is shown in Figure 3. The evaluator will enter the requested keyword in the 
column provided and press the "Search on Anapedia" button. The evaluator repeats the search process on 
Anapedia with different keywords. The evaluator then provides an assessment based on his experience 
interacting with the Anapedia website for a number of 83 Sirius framework criteria. 

















Figure 1. The Measurement model for evaluating Anapedia website 


Table 1. Aspects used in the Sirius framework evaluation 
Aspect Description 
General Aspects (GA) Components related to the site target, look & feel, coherence, and intensity of content revising. 
Indentity dan Information (I) | Components related to the sufficiency of the information architecture and site navigation. 
Structure dan Navigation (SN) | Components related to the site navigation and sufficiency of the information architecture 








Labelling (LB) Components related to the acquaintance of content labelling, importance and accuracy 

Layout of the Page (LY) Components related to the view and location of navigation and information components in the 
interface. 

Comprehensibility dan Components related to the quality and adequacy of icons, controls, and text contents of the 

Ease of Interaction (CI) interface. 

Control dan Feedback (CF) Components related to the navigation freedom, and information presented to the user in the 
interaction mechanism with the site. 

Multimedia Elements (ME) Components related to the multimedia contents to the site. 

Search (SE) Components related to the search facility applied in the site. 

Help (HE) Components related to the help feature prepared to the user while using the site. 
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Figure 2. Usability level measurement process using Sirius framework [13] 


Table 2. Assessment value of web usability 
Value Definition Numeric Value 
0..10 0: not conform at all, 10: fully conform 0, 1, 2, ..., 9, 10 
NWS Not conform in the whole site 0 








NML Not conform in the main links 2.5 
NHP Not sustained in the home page 5 

NSP Not conform in one or more subpages 73 
YES Fully conform 10 


NA Not applicable criteria - 








Figure 3. Anapedia homepage 


3. RESULTS AND ANALYSIS 
3.1. Calculating global usability 

In the third phase, a calculation is performed to determine the global usability value of the Anapedia 
web. Based on the content and objectives, the Anapedia website is included in the education web. Examples 
of global usability calculations from one evaluator for GA and II aspects can be seen in Table 3 (see in 
Appendix). The ev is the evaluation value given by the evaluators, sv is the conversion of the evaluator value, 
rv is the relevant criterion value for the educational web type, and wc is the weight coefficient of each 
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criterion. Then the final value for each criterion is calculated, these values are then added to the fav value. 
Based on the table, the global usability value of evaluator one is 94.66% (shown in Table 3 in the% usability 
column). The same steps are applied for the other six evaluators. Based on the results of the evaluations of all 
evaluators, an average global usability value of 81.4% was obtained which means that the usability of 
Anapedia's website is included in the high category [21]. Automatic evaluations for the Anapedia web are 
conducted using Google PageSpeed. Google PageSpeed can be used to measure the performance and loading 
pace of the web [17], [25]. Based on an analysis using Google PageSpeed, Anapedia's web loading speed is 
80% which is included in the HIGH category [26]. 


3.2. Analysis 

A comparison of the usability level of all aspects is shown in Figure 4. The results show that the best 
usability component on the Anapedia website is the labelling (LB) aspect of 84% where all the criteria in this 
aspect are considered GOOD. Labelling aspects include evaluating labels on the web, page title, and page 
uniform resource locator (URL). Conversely, the help (HE) aspect perform poorly with a value of 17%. This 
result is related to the absence of the Help feature on the Anapedia web. In addition to the two components, 
the usability criteria that has not been met by the Anapedia website and their weights are shown in Table 4. 
Table 5 presents a comparison of aspects of the Sirius framework with several quantitative evaluation methods. 


Table 4. Comparison of Sirius framework to existing methods 








Aspect Criteria Priority 
General Aspects AGI: Site goals are concrete and well defined MAJOR 
AG2: Offered content and services are precise and complete CRITICAL 
AG3: General site structure is user-oriented MAJOR 
AGé4: General Look & Feel matches the site goals, MODERATE 
characteristics, content and services 
AGS: Recognizable site visual design MODERATE 
AGO: Site's general design is coherent MAJOR 
AG8: Other languages are supported MODERATE 
AG9: Site's translation corrects and complete MODERATE 
Identity and Information I1: Identity or logotype is significant, identifiable MINOR 
and sufficiently visible 
IIS: Means of contact available MODERATE 
I6: Data Protection information is available for both MODERATE 


personal information and site content copyright 
Structure and Navigation SN3: Element organization is consistent with industry standards. MODERATE 


SN11: On image-based links, there is indication of MODERATE 
the content to be accessed. 
SN13: Elements that inform the user of where they MAJOR 
are and how to navigate back are present (e.g breadcrumbs) 
Page and Layout PLS: Visual space on the page is used correctly. MODERATE 
Multimedia Element ME6: Some added value is provided by using sound. MINOR 
Search SE6: Advanced search is provided. MODERATE 
Help HE1: Help link is located in a visible and expected location. MODERATE 
General Aspects HE2: Easy access to and return from the help system. MODERATE 
HE3: Contextual help is offered for complex tasks. MODERATE 
HE4: FAQ contain correct and relevant questions. MODERATE 
HES: FAQ contain correct and relevant answers. MODERATE 
AGI: Site goals are concrete and well defined MAJOR 





84% 
80% 75% 
71% 71% 
70% a 64% 
61% 60% 

60% 55% 
50% 
40% sabe 
30% 
20% 17% 

0% 

GA ul SN LB PL Ul CF ME SE HE 


Figure 4. Achievement level of Anapedia’s usability aspect 
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Table 5. Identified usability problems 








Elemen Evaluasi Heuristik Hasan et al. [20] Omar etal. [27] Nazir et al. [18] Torrente et al. [13] 
Number of aspect/category 3 5 3 10 
Number of criteria/metric 25 3T. 10 83 
Criteria weighting yes yes no yes 
Aspect/category weighting yes yes yes yes 
Evaluated web type specific specific any any 
Global usability calculation yes yes yes yes 
Used by expert no yes no yes 
Quantify non-compliance kriteria no no yes yes 
Quantify severity level yes yes no no 
Validation of the method no no no yes 





4. CONCLUSION 

This research provides information to the developer team regarding web quality in the context of 
usability for the improvement of design both in terms of interface and content. Evaluation by experts is one 
way to measure website usability. In this study we evaluated Anapedia Web, the first open web-based 
encyclopedia for children built in Bahasa, that is still considered low in terms of comfort. This aspect is 
related to the usability of the website. The automated and manual tools were conducted to measure the 
usability aspect of the app. Google PageSpeed app were used for the automatic evaluation and heuristic (with 
the involvement of a group of evaluators) for measuring the usability level manually. Based on the 
evaluations of all evaluators, the average value of global usability is considered high. To find out which 
aspects have the highest and lowest values, measurements are made based on evaluator evaluations and the 
best and worst aspects of the results are related to labeling and assistance. To be more thorough, it is 
necessary to do an assessment by the user using existing methods. 








APPENDIX 
Table 3. Calculating the GA dan II aspects of Anapedia for one evaluator 
Aspect Criteria Evaluation Sirius Relevance Weighting Final Criteria Final % Usability 
Value (ev) Value Value (rv) Coefficient Value Aspect (= fav x 10) 
(sv) (we) (sv x wc) (fav) 

General Site goals are 8 8 4 0.019417476 0.155339806 1,485 94.66 
aspect concrete and well 

defined 

Offered content 9 9 8 0.038834951 — 0.349514563 

and services are 

precise and 

complete 

General site 8 8 4 0.019417476 —0.155339806 

structure is user- 

oriented 

General Look & 9 9 2 0.009708738  0.087378641 

Feel matches the 

site goals, 

characteristics, 

content and 

services 

Recognizable site 8 8 2 0.009708738  0.077669903 

visual design 

Site's general 9 9 4 0.019417476 = 0.174757282 

design is coherent 

User language is S 10 4 0.019417476 = 0.194174757 

used throughout 

Other languages S 10 2 0.009708738 0.097087379 

are supported 

Site's translation S 10 2 0.009708738 0.097087379 

correct and 

complete 

Regularly updated S 10 2 0.009708738 0.097087379 


site 
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Table 3. Calculating the GA dan II aspects of Anapedia for one evaluator (continue) 








Aspect Criteria Evaluation Sirius Relevance Weighting Final Final % Usability 
Value (ev) Value Value (rv) Coefficient Criteria Aspect (2 fav x 10) 
(sv) (we) Value (fav) 
(sv x wc) 
Identity and Identity or 10 10 1 0.004854369 0.048543689 0.72815534 
Information logotype is 
significant, 
identifiable and 
sufficiently 
visible 
Site identity on S 10 2 0.009708738 — 0.097087379 
all pages 
Slogan or tagline 10 10 2 0.009708738 — 0.097087379 


is appropriate for 

the site goals. 

Information about S 10 2 0.009708738 0.097087379 
the site and 

company is 

available 

Means of contact S 10 2 0.009708738 0.097087379 
available 

Data Protection S 10 2 0.009708738 0.097087379 
information is 

available for both 

personal 

information and 

site content 

copyright 

Information S 10 4 0.019417476 0.194174757 
available about 

the author, 

sources, and 

content 

publishing and 

update dates. 
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